Today's networked computing environments are used in businesses for generating and storing large amounts of critical data. The systems used for moving, storing, and manipulating this critical data are expected to have high performance, high capacity, and high reliability, while being reasonably priced.
As is known in the art, large computer systems and data servers sometimes require large capacity data storage systems. One type of data storage system is a magnetic disk storage system. Here a bank of disk drives and the computer systems and data servers are coupled together through an interface. The interface includes storage processors that operate in such a way that they are transparent to the computer. That is, data is stored in, and retrieved from, the bank of disk drives in such a way that the computer system or data server merely thinks it is operating with one memory. One type of data storage system is a RAID data storage system. A RAID data storage system includes two or more disk drives in combination for fault tolerance and performance.
One conventional data storage system includes two storage processors for high availability. Each storage processor includes a respective send port and receive port for each disk drive. Accordingly, if one storage processor fails, the other storage processor has access to each disk drive and can attempt to continue operation.
Modern computer systems typically use a computer architecture that may be viewed as having three distinct subsystems which when combined, form what most think of when they hear the term computer. These subsystems are: 1) a processing complex; 2) an interface between the processing complex and I/O controllers or devices; and 3) the I/O (i.e., input/output) controllers or devices themselves. A processing complex may be as simple as a single microprocessor, such as a Pentium microprocessor, coupled to memory. Or, it might be as complex as two or more processors which share memory.
A blade server is essentially a processing complex, an interface, and I/O together on a relatively small printed circuit board that has a backplane connector. The blade is made to be inserted with other blades into a chassis that has a form factor similar to a rack server today. Many blades can be located in the same rack space previously required by just one or two rack servers. Blade servers typically provide all of the features of a pedestal or rack server, including a processing complex, an interface to I/O, and I/O. Further, the blade servers typically integrate all necessary I/O because they do not have an external bus which would allow them to add other I/O on to them. So, each blade typically includes such I/O as Ethernet (10/100, and/or 1 gig), and data storage control (SCSI, Fiber Channel, etc.).
The interface between the processing complex and I/O is commonly known as the Northbridge or memory control hub (MCH) chipset. On the “north” side of the chipset (i.e., between the processing complex and the chipset) is a bus referred to as the HOST bus. The HOST bus is usually a proprietary bus designed to interface to memory, to one or more microprocessors within the processing complex, and to the chipset. On the “south” side of the chipset are a number of buses which connect the chipset to I/O devices. Examples of such buses include: ISA, EISA, PCI, PCI-X, and Peripheral Component Interconnect (PCI) Express.
PCI Express is an I/O interconnect architecture that is intended to support a wide variety of computing and communications platforms and is described in the PCI Express Base Specification, Rev. 1.0a, Apr. 15, 2003 (hereinafter, “PCI Express Base Specification” or “PCI Express standard”). The PCI Express architecture describes a fabric topology in which the fabric is composed of point-to-point links that interconnect a set of devices. For example, a single fabric instance (referred to as a “hierarchy”) can include a Root Complex (RC), multiple endpoints (or I/O devices) and a switch. The switch supports communications between the RC and endpoints, as well as peer-to-peer communications between endpoints.
The switch includes a number of ports, with at least one port being connected to the RC and at least one other port being coupled to an endpoint as provided in the PCI Express Base Specification. The RC, switch, and endpoints may be referred to as “PCI Express devices”.
The switch may include ports connected to non-switch ports via corresponding PCI Express links, including a link that connects a switch port to a root complex port. The switch enables communications between the RC and endpoints, as well as peer-to-peer communications between endpoints. A switch port may be connected to another switch as well.
At least some of the end points may share an address domain, such as a memory address domain or an I/O address domain. The term “address domain” means the total range of addressable locations. If the shared address domain is a memory address domain, then data units are transmitted via memory mapped I/O to a destination address into the shared memory address domain. There may be more than two address domains, and more than one address domain may be shared. The address domains are contiguous ranges. Each address domains is defined by a master end point. Address portions associated with the individual end points may be non-contiguous and the term “portions” is meant to refer to contiguous and non-contiguous spaces. The master end point for a given address domain allocates address portions to the other end points which share that address domain. The end points communicate their address space needs to a master device, and the master device allocates address space accordingly.
Data units may be directed to one or more of the end points by addressing. That is, a destination address is associated with and may be included in the data units. The destination address determines which end point should receive a given data unit. Thus, data units addressed to the individual portion for a given end point should be received only by that end point. Depending on the embodiment, the destination address may be the same as the base address or may be within the address portion.
The end points may be associated with respective ports. Through this association, a given end point may send data units to and receive data units from its associated port. This association may be on a one-to-one basis. Because of these relationships, the ports also have associations with the address portions of the end points. Thus, the ports may be said to have address portions within the address domains.
A goal of PCI Express is to provide a migration strategy to expand from the legacy PCI technology into the new serial-based link technology. In at least some respects PCI Express aids this by being compatible to existing PCI hardware and software architectures. As a result, PCI Express also inherits a global memory address-based and tree topology architecture, which therefore is used in peer-to-peer communications between multiple hosts in various topologies, such as star, dual-star, and meshes, which are typically used in blade servers, clusters, storage arrays, and telecom routers and switches.
The PCI Express architecture is based upon a single host processor or root complex that controls the global memory address space of the entire system. Upon power-up and enumeration process, the root complex interrogates the entire system by traversing through the hierarchical tree-topology and locates all endpoint devices that are connection in the system. A space is allocated for each endpoint device in the global memory in order for the host processor to communicate with it.
In particular, a discovery and configuration cycle is begun in which each switch port and endpoint within the hierarchy is identified. The cycle comprises accessing configuration data stored in the each device coupled to the network switch fabric (e.g., the PCI configuration space of a PCI device). The switches comprise data related to devices that are coupled to the switch. If the configuration data regarding other devices stored by each switch is not complete, additional configuration cycles may be initiated until all devices coupled to the switches have been identified and the configuration data within each switch is complete.
During the aforementioned discovery and configuration operations, information is collected about each device installed in the system. Each PCI Express device stores information about its various device attributes, including capabilities and/or services supported by the device. The attribute information identifies functionality that may be accessed by the PCI Express device, such as mass storage or communication capabilities (via corresponding protocol interfaces), for example. The attributes parameter set (e.g., one or more attribute parameters in a list) is used, in part, to specify capabilities a requesting endpoint would like to access.
The attribute information may be stored in a table structure and may include a device ID, a vendor ID, a class code, a revision ID, a subsystem ID, a subsystem vendor ID, a capability pointer, and various reserved fields. The device ID comprises a 16-bit value assigned by the manufacturer of the device. The vendor ID is a 16-bit value assigned by PCI-SIG (Peripheral Component Interconnect Special Interest Group) for each vendor that manufacturers PCI Express-compliant devices. The class code is a 24-bit value that indicates the class of the device, as defined by PCI-SIG. The subsystem ID and subsystem vendor ID are analogous to the device ID and vendor ID, except they are applicable for devices that include PCI-compliant subsystems.
During link training, each PCI Express link is set up following a negotiation of link widths, frequency of operation and other parameters by the ports at each end of the link.
Fibre Channel is a high performance, serial interconnect standard designed for bi-directional, point-to-point communications between servers, storage systems, workstations, switches, and hubs. It offers a variety of benefits over other link-level protocols, including efficiency and high performance, scalability, simplicity, ease of use and installation, and support for popular high level protocols.
The Fibre Channel protocol (“FCP”) uses a single Open-Systems-Interface-like (OSI-like) stack architecture. Devices that are operable with the Fibre Channel protocol typically include a controller (an “FC controller”) that embodies the functionality of some of the middle-layers of the FCP stack. Furthermore, FC controllers may involve a “controller chip”. As part of the middle-layer FCP functionality, these FC controllers monitor the state of information transmissions over the FC communication links and are designed to take appropriate recovery measures should an unresponsive communication link be encountered.
A typical type of computer system test calls for the processor to execute firmware/software that operates at a lower level than an operating system based program, prior to booting the operating system. These include basic I/O system (BIOS) and power on self test (POST) programs. These types of tests provide relatively low-level control of component functionality and interconnect buses. BIOS and/or POST may use management signal connections (e.g., serial I2C (Inter-IC Bus) connections) to communicate with system components.